Add support for administrative operations in Blob adapter #3104

KodaiD · 2025-10-31T02:32:27Z

Description

This PR adds the Admin implementation for Blob adapter, enabling administrative operations over Azure Blob storage.

Support for S3 and GCS, and Storage operations will be addressed in subsequent PRs.

Related issues and/or PRs

N/A

Changes made

Added the DistributedStorageAdmin implementation for Object Storage.
Added the ObjectStorageWrapper implementation for Azure Blob storage.
Added unit tests.
Added integration tests.

Checklist

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes.
I have considered whether similar issues could occur in other products, components, or modules if this PR is for bug fixes.
Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
Tests (unit, integration, etc.) have been added for the changes.
My changes generate no new warnings.
Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

N/A

Release notes

Added support for administrative operations over Azure Blob Storage.

KodaiD · 2025-10-31T03:47:28Z

/gemini review

gemini-code-assist

Code Review

This pull request adds administrative operation support for the Azure Blob storage adapter. The changes are extensive, including new admin implementations, configurations, utility classes, and a suite of integration and unit tests.

The overall implementation looks solid and follows existing patterns in the codebase. I've found a few areas for improvement:

A potential bug in the test utilities where teardown logic might fail if certain resources don't exist.
A violation of the immutability contract in one of the new metadata classes.
A minor code style issue with redundant formatting.

Details are in the specific comments. Otherwise, the changes are well-structured and the test coverage is good.

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageTableMetadata.java

...c/integration-test/java/com/scalar/db/storage/objectstorage/ObjectStorageAdminTestUtils.java

core/src/main/java/com/scalar/db/storage/objectstorage/blob/BlobWrapper.java

KodaiD · 2025-10-31T04:31:40Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for administrative operations for the Azure Blob storage adapter. The changes are well-structured, with clear separation of concerns between the generic object storage logic and the specific Blob storage implementation. The use of a wrapper interface for the storage client is a good design choice. I've included a couple of comments for potential improvements, one regarding the robustness of a delete operation and another about handling concurrent administrative operations.

core/src/main/java/com/scalar/db/storage/objectstorage/blob/BlobWrapper.java

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageAdmin.java

KodaiD · 2025-10-31T04:43:26Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces administrative operation support for the Azure Blob storage adapter. The changes include a new ObjectStorageAdmin implementation that manages table and namespace metadata as objects within the storage. It also adds a generic ObjectStorageWrapper interface to abstract away the specifics of different object storage services, with an initial implementation for Azure Blob. The changes are well-structured and include comprehensive unit and integration tests. I have a couple of suggestions to reduce code duplication in the ObjectStorageAdmin class for better maintainability.

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageAdmin.java

Copilot

Pull Request Overview

This PR introduces Azure Blob Storage support for ScalarDB by implementing a new object storage adapter layer. The implementation provides admin functionality for metadata management without supporting full distributed storage operations.

Key changes:

Adds Azure Blob Storage wrapper and configuration classes with comprehensive storage operations
Implements ObjectStorageAdmin for managing table and namespace metadata stored in JSON format
Adds extensive test coverage including unit tests and integration tests with Azurite emulator

Reviewed Changes

Copilot reviewed 35 out of 36 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
core/build.gradle	Adds Azure Blob SDK dependency and new integration test source set
build.gradle	Adds Azure Blob Storage version configuration
.github/workflows/ci.yaml	Adds CI workflow for Azure Blob integration tests using Azurite emulator
BlobConfig.java	Configuration class parsing blob storage connection details and parallel upload settings
BlobWrapper.java	Azure Blob Storage implementation with CRUD operations and ETags for versioning
BlobProvider.java	Service provider interface implementation for blob storage
ObjectStorageAdmin.java	Admin operations managing metadata as JSON objects in blob storage
ObjectStorageUtils.java	Utility methods for constructing object keys and configuration
Serializer.java	JSON serialization/deserialization using Jackson
ConcatenationVisitor.java	Visitor pattern implementation for building concatenated partition keys
Multiple test files	Comprehensive unit and integration test coverage
ConsensusCommitSpecificIntegrationTestBase.java	Makes several test methods public for better testability
CoreError.java	Adds error codes for unsupported object storage operations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

core/src/main/java/com/scalar/db/storage/objectstorage/blob/BlobProvider.java

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageWrapperFactory.java

core/src/main/java/com/scalar/db/storage/objectstorage/blobstorage/BlobStorageWrapper.java

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageUtils.java

.github/workflows/ci.yaml

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageTableMetadata.java

core/src/test/java/com/scalar/db/storage/objectstorage/blobstorage/BlobStorageConfigTest.java

...va/com/scalar/db/transaction/consensuscommit/ConsensusCommitSpecificIntegrationTestBase.java

...rc/integration-test/java/com/scalar/db/storage/objectstorage/BlobWrapperIntegrationTest.java

komamitsu

LGTM! 👍

brfrn169

Left several comments. PTAL!

...n-test/java/com/scalar/db/storage/objectstorage/ObjectStorageWrapperIntegrationTestBase.java

core/src/main/java/com/scalar/db/storage/objectstorage/blobstorage/BlobStorageConfig.java

core/src/test/java/com/scalar/db/storage/objectstorage/BlobStorageConfigTest.java

core/src/main/java/com/scalar/db/storage/objectstorage/blobstorage/BlobStorageWrapper.java

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageAdmin.java

josh-wong

I've added some comments and suggestions. PTAL!

.github/workflows/ci.yaml

...om/scalar/db/storage/objectstorage/ConsensusCommitAdminIntegrationTestWithObjectStorage.java

core/src/main/java/com/scalar/db/common/CoreError.java

Torch3333

LGTM, thank you!

josh-wong

I've left some minor suggestions. Other than that, LGTM. Thank you!🙇🏻‍♂️

...om/scalar/db/storage/objectstorage/ConsensusCommitAdminIntegrationTestWithObjectStorage.java

...va/com/scalar/db/storage/objectstorage/ObjectStorageAdminCaseSensitivityIntegrationTest.java

...gration-test/java/com/scalar/db/storage/objectstorage/ObjectStorageAdminIntegrationTest.java

...orage/objectstorage/SingleCrudOperationTransactionAdminIntegrationTestWithObjectStorage.java

feeblefakie

LGTM! Thank you!

brfrn169

LGTM! Thank you!

Co-authored-by: Toshihiro Suzuki <[email protected]>

KodaiD added 7 commits October 22, 2025 20:45

Add Blob Storage adapter

90253bc

Merge branch 'master' into add-blob-adapter

71345d6

Refactor

9299df6

Add integration tests for Wrapper

8938ee9

Refactor

ba7b0e9

Merge branch 'master'

a60156d

Discard the unnecessary change

d5bf8ce

KodaiD self-assigned this Oct 31, 2025

KodaiD added the enhancement New feature or request label Oct 31, 2025

Fix workflow

e4cfa86

gemini-code-assist bot reviewed Oct 31, 2025

View reviewed changes

Apply suggestions

02a6708

gemini-code-assist bot reviewed Oct 31, 2025

View reviewed changes

core/src/main/java/com/scalar/db/storage/objectstorage/blob/BlobWrapper.java Outdated Show resolved Hide resolved

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageAdmin.java Outdated Show resolved Hide resolved

Apply suggestion

d2fb32f

gemini-code-assist bot reviewed Oct 31, 2025

View reviewed changes

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageAdmin.java Show resolved Hide resolved

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageAdmin.java Show resolved Hide resolved

Apply suggestions

ca8b0b6

KodaiD marked this pull request as ready for review October 31, 2025 04:53

Copilot AI review requested due to automatic review settings October 31, 2025 04:53

Copilot AI reviewed Oct 31, 2025

View reviewed changes

KodaiD requested review from a team, Torch3333, brfrn169, feeblefakie and komamitsu and removed request for a team October 31, 2025 07:14

KodaiD commented Oct 31, 2025

View reviewed changes

...va/com/scalar/db/transaction/consensuscommit/ConsensusCommitSpecificIntegrationTestBase.java Show resolved Hide resolved

KodaiD commented Oct 31, 2025

View reviewed changes

...rc/integration-test/java/com/scalar/db/storage/objectstorage/BlobWrapperIntegrationTest.java Outdated Show resolved Hide resolved

KodaiD added 2 commits November 4, 2025 21:01

Apply suggestions

ed60372

Refactor BlobStorageConfigTest

1c20822

KodaiD requested a review from feeblefakie November 4, 2025 12:09

Remove metadata versioning

fb4ab41

komamitsu approved these changes Nov 5, 2025

View reviewed changes

brfrn169 requested a review from josh-wong November 5, 2025 05:42

brfrn169 reviewed Nov 5, 2025

View reviewed changes

KodaiD added 3 commits November 5, 2025 17:10

Apply suggestions

df628e5

Merge remote-tracking branch 'origin/master' into add-blob-admin

c26d52d

Resolve conflicts

4e647c7

KodaiD requested a review from brfrn169 November 5, 2025 09:14

josh-wong requested changes Nov 5, 2025

View reviewed changes

Apply suggestions

38e72c3

KodaiD requested a review from josh-wong November 5, 2025 13:06

Torch3333 approved these changes Nov 6, 2025

View reviewed changes

josh-wong approved these changes Nov 6, 2025

View reviewed changes

Apply suggestions

63847cc

feeblefakie approved these changes Nov 6, 2025

View reviewed changes

Merge branch 'master' into add-blob-admin

c1db126

brfrn169 reviewed Nov 6, 2025

View reviewed changes

brfrn169 approved these changes Nov 6, 2025

View reviewed changes

KodaiD merged commit b3d44a1 into master Nov 6, 2025
131 of 132 checks passed

KodaiD deleted the add-blob-admin branch November 6, 2025 08:30

feeblefakie pushed a commit that referenced this pull request Nov 6, 2025

Add support for administrative operations in Blob adapter (#3104)

58c7787

Co-authored-by: Toshihiro Suzuki <[email protected]>

feeblefakie mentioned this pull request Nov 6, 2025

Backport to branch(3) : Add support for administrative operations in Blob adapter #3123

Merged

This was referenced Nov 6, 2025

Add support for data manipulation operations in Blob Storage adapter #3124

Merged

Add validation for secondary indexes in table metadata with Object Storage #3127

Merged

Add support for S3 #3141

Merged

KodaiD mentioned this pull request Nov 18, 2025

Add support for Cloud Storage #3179

Merged

7 tasks

KodaiD mentioned this pull request Nov 26, 2025

Add setup for Blob Storage scalar-labs/scalardb-samples#84

Open

6 tasks

Add support for administrative operations in Blob adapter #3104

Add support for administrative operations in Blob adapter #3104

Uh oh!

Conversation

KodaiD commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues and/or PRs

Changes made

Checklist

Additional notes (optional)

Release notes

Uh oh!

KodaiD commented Oct 31, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KodaiD commented Oct 31, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

KodaiD commented Oct 31, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

komamitsu left a comment

Choose a reason for hiding this comment

Uh oh!

brfrn169 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

josh-wong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Torch3333 left a comment

Choose a reason for hiding this comment

Uh oh!

KodaiD commented Oct 31, 2025 •

edited

Loading